The state of ILOs - Report
Sofia Vazquez Alferez (DARS, University College Maastricht
08/07/2019
Abstract
This document presents a brief overview of the current state of the Intended Learning Objectives (hereafter ILO). Including information on:
The number of ILOs in our courses.
The orientation of language used in ILOs
The most common capacities the ILOs promote broken down by Level and Concentration
[In progress] A description of how the current objectives map to Bloom’s Taxonomy of learning.
[NOT DONE YET] Recommendations on how to write ILOs
[NOT DONE YET] Suggested rephrasing of current ILO’s
A brief overview of our data
For this analysis I used the ILOs for the year 2018/2019 provided by Edith. The ILOs of the projects (PRO) were missing from this file, and have therefore not been included in the subsequent analysis. (I did have the ILOs for the Undergraduate Research Projects (UGR), so these are inlcuded).
The data contained a total of 142 courses, which amounted to 625 ILOs. All quotes from ILOs are referenced with base to the ILOs_2018_2019.docx document.
(It is possible to add some insight into the ILOs of projects and Capstone by using data from previous years. However, this old data does not include any of the ILOs for the new research projects, nor some of the ILOs for older projects, such as Deep Reading. The file is called “Missing_ILOs.xlsx” and is available for integration in the present report upon request).
How many of ILOs do our courses have? (distribution per course)
In general, the courses had an average of 4.4 ILOs per course. Which were distributed as follows:
A breakdown by Level and Concentration gives:
##With what orientation are they written? (Student vs. Course) It was possible to distinguish two categories in the way ILOs were formulated. ILOs were either as student (S) oriented or course (C) oriented. Student oriented ILOs described what the student was supposed to achieve or have learned during the course, whilst course oriented ILOs described the aims of the course. For example:
- Student Oriented ILO:
“To offer a broad overview of scientific models” (COR1005, p. 1)
“To acquaint students with the problems…” (HUM2030, p.3)
“To enhance their research skills” (SKI1009, p.11)
- Course Oriented ILO:
“Apply basic bookkeeping techniques” (SSC2022, p.15)
“To reflect on the relevance and utility of social theory in general”(SSC2028, p.15)
A few courses had some ILOs formulated in a student oriented fashion and others ILOs in a course oriented fashion. Thus, overview bellow is done at a granularity of ILO not courses.
In general, this is what we have in the curriculum in raw numbers:
In percentages, we have the following:
###Inspecting NA orientation category The NA category represents courses that I did not classify in any of those categories. They were the following ILOs:
| Course | ILO |
|---|---|
| SCI2009 | 1. Human Cells and functions |
| SCI2009 | 2. Functional organization of the body |
| SCI2009 | 3. Membrane Physiology |
| SCI2009 | 4. Cardiac function and blood pressure control |
| SCI2009 | 5. Red blood cell function and gas transport and exchange |
| SCI2009 | 6. Pulmonary ventilation and regulation |
| SCI2009 | 7. Kidney function, intra-and extracellular compartments |
| SCI2009 | 8. Neuronal control |
| SCI2009 | 9. Hormonal control |
| SCI2009 | 10. Gastrointestinal Physiology |
| SCI2010 | 1. Fairness and cooperation - in the chapters on Cooperative Games and Bankruptcy Problems |
| SCI2010 | 2. Rationality and Common Knowledge - in the chapters on Extensive Form Games, Normal Form Games, Matrix Games, Repeated Games and Repeated Games with Absorbing States |
| SCI2010 | 3. Threats and Manipulations - in the part on Repeated Games |
| SCI2010 | 4. Expectations - in the chapters on Normal Form Games, Matrix Games, Repeated Games and Repeated Games with Absorbing States |
| SCI2010 | 5. Nonmanipulability - in the chapter on Mecanism Design |
| SCI2031 | 1. Cells and humoral factors of the innate and adaptive immune system. |
| SCI2031 | 2. Cellular and molecular effector mechanisms of the innate and adaptive immunity during inflammation and infection. |
| SCI2031 | 3. The structure and function of primary and secondary lymphoid tissue. |
| SCI2031 | 4. The processes in the immune response after immunisation and vaccination. |
| SCI2031 | 5. Immune mechanisms in disease. |
| SSC1009 | All previous 5 objectives |
| SSC2052 | NA |
| SSC3030 | 1. The legal foundations of the European Union |
| SSC3030 | 2. The institutions of the EU, their historical evolution and the horizontal relationship between them |
| SSC3030 | 3. The vertical relationship between the EU and the Member States including the principles of conferral, supremacy, subsidiarity, proportionality and loyalty |
| SSC3030 | 4. The implementation and enforcement mechanisms of EU law infringement proceedings against Member States, enforcement through national courts, legal review of EU action. |
| SSC3057 | Standard economic paradigm of expected utility theory and rational choice |
| SSC3057 | What is the influence of neurology, psychology and sociology on the economic paradigm |
| SSC3057 | Development of macroeconomics and policies |
| NA | NA |
Analysing our ILOs
Methodology
Phase 1- Verb extraction
In order to get a better grasp of the characterization of our ILOs, I manually went through all ILOs and extracted the verb(s) describing what the student was supposed to do in each ILO. This created a distiction between the course verb and the student verb, as well as between the action verb and the intended verb. For instance:
1. Course verb/Student verb:
For the ILO “to provide students with […] perspectives to examine…” (COR1004, p.1) the extracted verb is “[to] examine” not “[to] provide”.
2. Action verb/Intended verb: For the ILO “to have the ability to interpret dynamical phenomena…”(SCI3006, p. 9) the verb “[to] interpret” was extracted not “[to] have (the ability)”.
In cases where the verb used was missleading, descriptive words were included. For instance, in the ILO “Gain basic knowledge in using economic/statistical data and present them in an informative way” (SSC2038, p. 15), the words “Gain basic knowledge” were recorded, as opposed to simply “[to] gain”. For this specific ILO, the verbs “use” and “present” were also extracted, as each ILO could have more than one verb associated to them.
During this phase I tried to keep as close as possible to the original formulation, although as I advanced through the ILOs I started to adapt some of the formulations to create some consistency with previously encountered data. Thus, particularily at the end formulations such as “to perform an analysis” were simply extracted as “analyse”. Moreover, because of this fidelity principle, some of the verbs for ilos were not verbs at all but conveyed the expected outcome. For instance, “overview” was extracted from the following ILO: “To give an overview over the different media platforms and media practices” (HUM2022, p.3), since the student was suppoded to ‘get an overview’.
This phase was performed in Excel and all extracted verbs were recorded in the same row as the original formulation, so it is possible to trace back the work and contest my check my interpretations.
In the following list, it is possible to see all the verbs that were extracted:
| Verbs |
|---|
| to know |
| to have knowledge |
| to understand |
| to develop attitude |
| to develop understanding |
| basic understanding |
| get overview |
| work with models |
| model |
| acquaint |
| familiarize |
| introduce |
| examine |
| develop own analysis |
| [get] notion |
| understand |
| evaluate |
| [get] introduction |
| explore |
| test |
| integrate |
| analyse |
| express |
| [gain] familiarity |
| to write |
| [get] overview |
| highlight |
| pinpoint characteristics |
| explain |
| apply |
| set up |
| write |
| get acquainted |
| gain insight |
| basic introduction |
| explore the meaning |
| explore how |
| to study |
| critical reflection |
| learn |
| to trace |
| reflect |
| gain understanding |
| basic knowledge |
| select |
| communicate |
| demonstrate awareness |
| recognize |
| view films critically |
| [get] showned |
| NA |
| close reading |
| identify |
| develop sensitivity |
| distinguish |
| trace |
| to look at |
| construct design |
| address the what, who, why, how, when, where |
| study |
| present |
| grasp |
| critically analyse |
| contribute to debates |
| develop own understaning |
| describe |
| take position in debate |
| gain sufficient background |
| be presented with |
| appreciate (more) |
| acquire basic toolbox |
| reason qualitatively |
| be prepared |
| notice |
| elaborate |
| give examples |
| point out |
| design |
| use |
| improve problem solving skills |
| solve |
| develop computational skills |
| know |
| gain basic practical knowledge |
| obtain basic knowledge |
| review |
| cast |
| make use |
| interpret |
| presentation skills |
| meet |
| collect |
| demonstrate |
| translate |
| establish (link) |
| enrich discussion |
| propose (solutions) |
| work |
| formulate |
| produce |
| interview |
| execute |
| conduct |
| practice |
| enhance research skills |
| experience |
| brain storm |
| deliver |
| integrate (visual aids) |
| give feedback |
| carve out (underlying structure) |
| build argument |
| plan |
| make coherent |
| gain expertise |
| perform |
| report |
| find |
| prepare |
| position interest |
| work together |
| transcribe |
| be engaged (in scientific inquiry) |
| gain funcionalist vision |
| judge |
| comprehend |
| differentiate |
| coduct |
| make contact |
| discover |
| read |
| construct |
| engage in socio-legal thinking |
| be conversant |
| survey |
| develop |
| see |
| consider |
| retreive |
| list |
| provide (reasons) |
| form reasoned opinions |
| become aware |
| research |
| discuss |
| develop (an approach to) |
| retain |
| investigate |
| reflect (critically) |
| (conduct) research |
| cope |
| work in groups |
| keep infromed |
| frame |
| approach |
| view |
| assess |
| appreciate |
| value |
| gain perspective |
| deal with |
| discern |
| peer reviewing skills |
| to be able to see |
| use general models and modelling techiques |
| compare |
| connect |
| get an idea of |
| situate in context |
| debate |
| to explain |
| use specialized terms |
| reconstruct |
| (apply)adopt method |
| execute design |
| contextualize |
| think practically |
| talk |
| reason analyticaly |
| compose |
| debug |
| define |
| reason academically |
| inspire |
| pitch |
| organise |
| adapt |
| avoid |
| modify |
| gather |
| turn into |
| set up plan |
| observing |
| create |
| understand (critically) |
| manage |
| unlock (ongoing debates) |
| speak (“orally”) |
| illustrate |
| weigh |
| retain content |
| extract |
| give (opinion) |
| display (tolerance) |
| use frameworks |
| incorporate feedback |
| to put trends in context |
| synthesise and explain |
| to discuss |
| apply criticism |
| run programs |
| answer |
| outline |
| interact |
| speak |
| choose |
| taking fieldnotes & interviewing |
| reduce (stereotypes) |
| write/plead |
| define/analyse/answer |
| criticize |
| to further (research, analyical and writing skills) |
| reinforce (opinion) |
| use knowledge |
Phase 2- Standardisation
As you can see, some verbs are really similar. For example, we have: “to understand”, “understand” and “basic understanding”. Therefore, in Phase 2 I standardised some of the vocabulary. All of the previous words were replaced by the same words: “understand”. Here is an overview of the replacements:
## [1] "The verbs that were taken to be the same as \"understand\" are: to understand, to develop understanding, basic understanding, understand, gain understanding, develop own understaning, understand (critically)"## [1] "The verbs that were taken to be the same as \"know\" are: to know, to have knowledge, basic knowledge, know, obtain basic knowledge"## [1] "The verbs that were taken to be the same as \"[gain] familiarity\" are: familiarize, [gain] familiarity"## [1] "The verbs that were taken to be the same as \"analyse\" are: develop own analysis, analyse, critically analyse"## [1] "The verbs that were taken to be the same as \"[be] introduced\" are: introduce, [get] introduction, basic introduction"## [1] "The verbs that were taken to be the same as \"[get] overview\" are: get overview, [get] overview"## [1] "The verbs that were taken to be the same as \"[get] acquainted\" are: acquaint, get acquainted"The result is a table like this:
| Course | ILO | Verb |
|---|---|---|
| COR1002 | To have knowledge of a number of specific problems in the foundations of the social sciences, such as explanation vs understanding, structuralist vs individualist views on explaining human behavior, the limits of prediction in the social sciences, the role of social science in society | know |
| COR1002 | To have knowledge of the major problems or topics in the philosophy of science, such as the demarcation between science and non-science, the role of observation in science, the nature of scientific method notably induction and falsification, the issues of realism and instrumentalism, the problem of progress in knowledgethe difficulties regarding causality in history | know |
| COR1002 | To know the major approaches in the philosophy of science, such as the traditional or received view, Karl Poppers critical rationalism and its variants, Kuhns theory of scientific revolutions | know |
| COR1003 | To Develop a critical understanding concerning the relation between perspective bias, facts, and context, as well as the difficulties regarding causality in history. | understand |
| COR1003 | To develop a critical attitude towards the interpretation of historical data and processes | develop attitude |
| COR1003 | To understand the main trends in politics, demography, society and culture since 1945, and will be able to see and put these trends in a global context. | understand |
| COR1003 | To understand the main trends in politics, demography, society and culture since 1945, and will be able to see and put these trends in a global context. | be able to see |
| COR1003 | To understand the main trends in politics, demography, society and culture since 1945, and will be able to see and put these trends in a global context. | put trends in context |
| COR1004 | To provide the students with a basic understanding of what political philosophy is about and why it is important and useful in understanding contemporary democratic societies. | understand |
| COR1004 | To understand the central concepts like justice and equality in theory, and in application. | understand |
| COR1005 | To offer a broad overview of scientific models and modelling techniques in different disciplines | [get] overview |
| COR1005 | To teach students how to model a specific phenomenon by using general models and modelling techniques | model |
| COR1005 | To teach students how to model a specific phenomenon by using general models and modelling techniques | use general models and modelling techiques |
Results
Rough numeric overview:
After the previous methodology has been applied, we have 771 individual capacity learning objectives (hereafter capacities). These are the abilities we intend to promote in our students according to our learning objectives and correspond to individual “verbs” (e.g. “understand”, “analyse”). The breakdown by level is as follows:
Level 1000: 123 capacities
Level 2000: 387 capacities.
Level 3000: 123 capacities.
Graphically:
Across all courses:
By Level:
Capacities inspection
So what are these capacities?
Across all courses
Breakdown by Level
Breakdown by Concentration
Mapping to Blooms taxonomy
Methodology
Objective: To map the verbs used in the ILOs to the cognitive process dimension of Bloom’s Taxonomy.
Procedure:
The verbs the previoulsy identified and standardised verbs from the ILOs were extracted and put into a separate Excel file.
The verb was matched to a dimension on and mapped to Blooms dimensions using Bloom’s Taxonomy Verb Chart form the University of Arkansas (available here: https://tips.uark.edu/blooms-taxonomy-verb-chart/) as a guide.
The file was imported back and the outcome can be seen below.
Brief description of Bloom’s Taxonomy
In general learning objectives are understood as consisting of two parts, usually a verb and a noun, corresponding to the
“the kind of behavior to be developed in the student and the content . .. in which this behavior is to operate”(Tyler,1949, p. 30).
Roughly, in Bloom’s Taxonomy the behaviour or verb is refered to as the cognitive process. This is to promote an understandning of the verb in the learning objectives in terms of the cognitive activity of the student rather than their end behaviour. For instance, a learning objective of the kind “the student will be able to list…” implies (using Bloom) the simple act of remembering. In contrast, without emphasis on cognitive processes, a student might be required to analyse material from several sources and develop the list which they then communicate. This is far more complex than remembering and it is why the emphasis is made on cognitive activities to avoid ambiguity.
The levels of the congnitive process dimension in Bloom’s taxonomy are:
Remember - Retreive knowledge from long term memory.
Understand - Construct meaning from materials.
Apply - Carry out or use a procedure in a given situation.
Analyse - Break material into constituent parts, and determine the relation between parts to one another and to an overall structure or purpose
Evaluate - Make judgments based on criteria and standards
Create - Put elements together to form a coherent or functional whole; reorganize elements into a new pattern or structure.
(Anderson, Krathwohl & Bloom, 2001, “5.1 The cognitive process dimension”)
Results
Mapping the verbs used according to the capacity, we have:
| Verb | Remember | Understand | Apply | Analyse | Evaluate | Create |
|---|---|---|---|---|---|---|
| know | 1 | 1 | 0 | 0 | 0 | 0 |
| understand | 0 | 1 | 0 | 0 | 0 | 0 |
| develop attitude | 0 | 0 | 0 | 0 | 0 | 1 |
| [get] overview | 0 | 1 | 0 | 0 | 0 | 0 |
| work with models | 0 | 0 | 1 | 0 | 0 | 0 |
| model | 0 | 0 | 0 | 0 | 0 | 1 |
| [get] acquainted | 0 | 1 | 0 | 0 | 0 | 0 |
| [gain] familiarity | 0 | 1 | 0 | 0 | 0 | 0 |
| [be] introduced | 0 | 1 | 0 | 0 | 0 | 0 |
| examine | 0 | 0 | 0 | 1 | 0 | 0 |
| analyse | 0 | 0 | 0 | 1 | 0 | 0 |
| [get] notion | 0 | 1 | 0 | 0 | 0 | 0 |
| evaluate | 0 | 0 | 0 | 0 | 1 | 0 |
| [get] introduction | 0 | 1 | 0 | 0 | 0 | 0 |
| explore | 0 | 0 | 1 | 1 | 0 | 0 |
| test | 0 | 0 | 0 | 0 | 1 | 0 |
| integrate | 0 | 0 | 0 | 0 | 0 | 1 |
| express | 0 | 1 | 1 | 0 | 0 | 0 |
| write | 0 | 0 | 1 | 0 | 0 | 0 |
| highlight | 0 | 0 | 0 | 1 | 0 | 0 |
| pinpoint characteristics | 0 | 0 | 0 | 1 | 0 | 0 |
| explain | 0 | 1 | 0 | 1 | 1 | 0 |
| apply | 0 | 0 | 1 | 0 | 0 | 0 |
| set up | 0 | 0 | 1 | 0 | 0 | 0 |
| gain insight | 0 | 1 | 0 | 0 | 0 | 0 |
| explore the meaning | 0 | 0 | 0 | 1 | 0 | 0 |
| explore how | 0 | 0 | 0 | 1 | 0 | 0 |
| study | 0 | 0 | 0 | 1 | 0 | 0 |
| critical reflection | 0 | 0 | 0 | 0 | 1 | 0 |
| learn | 0 | 1 | 1 | 0 | 0 | 0 |
| trace | 0 | 0 | 1 | 0 | 0 | 0 |
| reflect | 0 | 0 | 0 | 0 | 1 | 0 |
| select | 0 | 0 | 0 | 0 | 1 | 0 |
| communicate | 0 | 1 | 0 | 0 | 0 | 0 |
| demonstrate awareness | 0 | 0 | 1 | 0 | 0 | 0 |
| recognize | 1 | 0 | 0 | 0 | 0 | 0 |
| view films critically | 0 | 0 | 0 | 1 | 1 | 0 |
| [get] showned | 1 | 0 | 0 | 0 | 0 | 0 |
| NA | 0 | 0 | 0 | 0 | 0 | 0 |
| close reading | 0 | 0 | 0 | 1 | 0 | 0 |
| identify | 1 | 0 | 0 | 0 | 0 | 0 |
| develop sensitivity | 0 | 0 | 1 | 0 | 0 | 0 |
| distinguish | 0 | 0 | 0 | 1 | 0 | 0 |
| look at | 0 | 1 | 0 | 0 | 0 | 0 |
| construct design | 0 | 0 | 0 | 0 | 0 | 1 |
| address the what, who, why, how, when, where | 0 | 1 | 0 | 0 | 0 | 0 |
| present | 0 | 0 | 1 | 0 | 0 | 0 |
| grasp | 0 | 0 | 1 | 0 | 0 | 0 |
| contribute debates | 0 | 0 | 0 | 0 | 0 | 0 |
| describe | 0 | 1 | 0 | 0 | 0 | 0 |
| take position in debate | 0 | 0 | 1 | 0 | 0 | 0 |
| gain sufficient background | 0 | 1 | 0 | 0 | 0 | 0 |
| be presented with | 0 | 1 | 0 | 0 | 0 | 0 |
| appreciate (more) | 0 | 0 | 0 | 1 | 0 | 0 |
| acquire basic toolbox | 0 | 0 | 1 | 0 | 0 | 0 |
| reason qualitatively | 0 | 0 | 0 | 1 | 0 | 0 |
| be prepared | 0 | 1 | 0 | 0 | 0 | 0 |
| notice | 1 | 0 | 0 | 0 | 0 | 0 |
| elaborate | 0 | 0 | 1 | 0 | 0 | 0 |
| give examples | 0 | 1 | 0 | 0 | 0 | 0 |
| point out | 0 | 1 | 0 | 0 | 0 | 0 |
| design | 0 | 0 | 0 | 0 | 0 | 1 |
| use | 0 | 0 | 1 | 0 | 0 | 0 |
| improve problem solving skills | 0 | 0 | 1 | 0 | 0 | 0 |
| solve | 0 | 0 | 1 | 0 | 0 | 0 |
| develop computational skills | 0 | 0 | 1 | 0 | 0 | 0 |
| gain basic practical knowledge | 0 | 0 | 1 | 0 | 0 | 0 |
| review | 0 | 1 | 0 | 0 | 0 | 0 |
| cast | 0 | 0 | 1 | 0 | 0 | 0 |
| make use | 0 | 0 | 1 | 0 | 0 | 0 |
| interpret | 0 | 1 | 0 | 0 | 0 | 0 |
| presentation skills | 0 | 0 | 1 | 0 | 0 | 0 |
| meet | 0 | 0 | 1 | 0 | 0 | 0 |
| collect | 0 | 0 | 1 | 0 | 0 | 0 |
| demonstrate | 0 | 0 | 0 | 1 | 0 | 0 |
| translate | 0 | 0 | 1 | 0 | 0 | 0 |
| establish (link) | 0 | 1 | 0 | 0 | 0 | 0 |
| enrich discussion | 0 | 0 | 1 | 0 | 0 | 0 |
| propose (solutions) | 0 | 0 | 1 | 0 | 0 | 0 |
| work | 0 | 0 | 1 | 0 | 0 | 0 |
| formulate | 0 | 1 | 0 | 0 | 0 | 0 |
| produce | 0 | 0 | 0 | 0 | 0 | 1 |
| interview | 0 | 0 | 1 | 0 | 0 | 0 |
| execute | 0 | 0 | 1 | 0 | 0 | 0 |
| conduct | 0 | 0 | 1 | 0 | 0 | 0 |
| practice | 0 | 0 | 1 | 0 | 0 | 0 |
| enhance research skills | 0 | 0 | 0 | 0 | 0 | 1 |
| experience | 0 | 0 | 1 | 0 | 0 | 0 |
| brain storm | 1 | 0 | 0 | 0 | 0 | 0 |
| deliver | 0 | 0 | 1 | 0 | 0 | 0 |
| integrate (visual aids) | 0 | 0 | 1 | 0 | 0 | 0 |
| give feedback | 0 | 0 | 0 | 0 | 1 | 0 |
| carve out (underlying structure) | 0 | 0 | 0 | 1 | 0 | 0 |
| build argument | 0 | 0 | 0 | 0 | 0 | 1 |
| plan | 0 | 0 | 0 | 0 | 0 | 1 |
| make coherent | 0 | 0 | 0 | 0 | 1 | 0 |
| gain expertise | 0 | 0 | 0 | 1 | 0 | 0 |
| perform | 0 | 0 | 1 | 0 | 0 | 0 |
| report | 0 | 0 | 0 | 1 | 0 | 0 |
| find | 0 | 0 | 0 | 0 | 1 | 0 |
| prepare | 0 | 0 | 1 | 0 | 0 | 0 |
| position interest | 0 | 0 | 0 | 0 | 1 | 0 |
| work together | 0 | 0 | 1 | 0 | 0 | 0 |
| transcribe | 0 | 0 | 0 | 0 | 0 | 0 |
| be engaged (in scientific inquiry) | 0 | 0 | 1 | 0 | 0 | 0 |
| gain funcionalist vision | 0 | 1 | 0 | 0 | 0 | 0 |
| judge | 0 | 0 | 0 | 0 | 1 | 0 |
| comprehend | 0 | 1 | 0 | 0 | 0 | 0 |
| differentiate | 0 | 0 | 0 | 0 | 1 | 0 |
| coduct | 0 | 0 | 1 | 0 | 0 | 0 |
| make contact | 0 | 0 | 1 | 0 | 0 | 0 |
| discover | 0 | 0 | 1 | 0 | 0 | 0 |
| read | 0 | 1 | 0 | 0 | 0 | 0 |
| construct | 0 | 0 | 0 | 0 | 0 | 1 |
| engage in socio-legal thinking | 0 | 0 | 0 | 0 | 1 | 0 |
| be conversant | 0 | 1 | 0 | 0 | 0 | 0 |
| survey | 0 | 1 | 0 | 0 | 0 | 0 |
| develop | 0 | 0 | 0 | 0 | 0 | 1 |
| see | 0 | 1 | 0 | 0 | 0 | 0 |
| consider | 0 | 0 | 0 | 1 | 0 | 0 |
| retreive | 1 | 0 | 0 | 0 | 0 | 0 |
| list | 1 | 0 | 0 | 0 | 0 | 0 |
| provide (reasons) | 0 | 1 | 0 | 0 | 0 | 0 |
| form reasoned opinions | 0 | 0 | 0 | 0 | 1 | 0 |
| become aware | 0 | 1 | 0 | 0 | 0 | 0 |
| research | 0 | 0 | 0 | 0 | 0 | 1 |
| discuss | 0 | 0 | 0 | 1 | 0 | 0 |
| develop (an approach to) | 0 | 0 | 0 | 0 | 0 | 1 |
| retain | 1 | 0 | 0 | 0 | 0 | 0 |
| investigate | 0 | 0 | 0 | 0 | 1 | 0 |
| reflect (critically) | 0 | 0 | 0 | 0 | 1 | 0 |
| (conduct) research | 0 | 0 | 0 | 0 | 0 | 1 |
| cope | 0 | 0 | 0 | 1 | 0 | 0 |
| work in groups | 0 | 0 | 1 | 0 | 0 | 0 |
| keep infromed | 1 | 0 | 0 | 0 | 0 | 0 |
| frame | 0 | 0 | 0 | 1 | 0 | 0 |
| approach | 0 | 0 | 1 | 0 | 0 | 0 |
| view | 0 | 1 | 0 | 0 | 0 | 0 |
| assess | 0 | 0 | 0 | 0 | 1 | 0 |
| appreciate | 0 | 0 | 0 | 0 | 1 | 0 |
| value | 0 | 0 | 0 | 0 | 1 | 0 |
| gain perspective | 0 | 1 | 0 | 0 | 0 | 0 |
| deal with | 0 | 0 | 1 | 0 | 0 | 0 |
| discern | 0 | 0 | 0 | 1 | 0 | 0 |
| peer reviewing skills | 0 | 0 | 0 | 0 | 1 | 0 |
| be able to see | 0 | 1 | 0 | 0 | 0 | 0 |
| use general models and modelling techiques | 0 | 0 | 1 | 0 | 0 | 0 |
| compare | 0 | 0 | 1 | 0 | 0 | 0 |
| connect | 0 | 1 | 0 | 0 | 0 | 0 |
| get an idea of | 0 | 1 | 0 | 0 | 0 | 0 |
| situate in context | 0 | 0 | 0 | 0 | 1 | 0 |
| debate | 0 | 0 | 1 | 0 | 0 | 0 |
| use specialized terms | 0 | 0 | 0 | 1 | 0 | 0 |
| reconstruct | 0 | 0 | 0 | 0 | 1 | 0 |
| (apply)adopt method | 0 | 0 | 1 | 0 | 0 | 0 |
| execute design | 0 | 0 | 1 | 0 | 0 | 0 |
| contextualize | 0 | 0 | 0 | 0 | 1 | 0 |
| think practically | 0 | 0 | 0 | 0 | 1 | 0 |
| talk | 0 | 1 | 0 | 0 | 0 | 0 |
| reason analyticaly | 0 | 0 | 0 | 1 | 0 | 0 |
| compose | 0 | 0 | 0 | 0 | 0 | 1 |
| debug | 0 | 0 | 1 | 0 | 0 | 0 |
| define | 1 | 0 | 0 | 0 | 0 | 0 |
| reason academically | 0 | 0 | 0 | 1 | 0 | 0 |
| inspire | 0 | 1 | 0 | 0 | 0 | 0 |
| pitch | 0 | 0 | 1 | 0 | 0 | 0 |
| organise | 0 | 0 | 1 | 0 | 0 | 0 |
| adapt | 0 | 0 | 0 | 0 | 1 | 0 |
| avoid | 0 | 0 | 1 | 0 | 0 | 0 |
| modify | 0 | 0 | 0 | 0 | 1 | 0 |
| gather | 1 | 0 | 0 | 0 | 0 | 0 |
| turn into | 0 | 0 | 1 | 0 | 0 | 0 |
| set up plan | 0 | 0 | 1 | 0 | 0 | 0 |
| observing | 0 | 1 | 0 | 0 | 0 | 0 |
| create | 0 | 0 | 0 | 0 | 0 | 1 |
| understand (critically) | 0 | 1 | 0 | 0 | 0 | 0 |
| manage | 0 | 0 | 1 | 0 | 0 | 0 |
| unlock (ongoing debates) | 0 | 1 | 0 | 0 | 0 | 0 |
| speak (“orally”) | 0 | 0 | 1 | 0 | 0 | 0 |
| illustrate | 0 | 1 | 1 | 0 | 0 | 0 |
| weigh | 0 | 0 | 0 | 0 | 1 | 0 |
| retain content | 1 | 0 | 0 | 0 | 0 | 0 |
| extract | 0 | 0 | 1 | 0 | 0 | 0 |
| give (opinion) | 0 | 1 | 0 | 0 | 0 | 0 |
| display (tolerance) | 0 | 0 | 1 | 0 | 0 | 0 |
| use frameworks | 0 | 0 | 1 | 0 | 0 | 0 |
| incorporate feedback | 0 | 0 | 1 | 0 | 0 | 0 |
| put trends in context | 0 | 0 | 0 | 0 | 1 | 0 |
| synthesise and explain | 0 | 1 | 0 | 0 | 0 | 0 |
| apply criticism | 0 | 0 | 1 | 0 | 0 | 0 |
| run programs | 0 | 0 | 1 | 0 | 0 | 0 |
| answer | 0 | 1 | 0 | 0 | 0 | 0 |
| outline | 0 | 1 | 0 | 0 | 0 | 0 |
| interact | 0 | 0 | 1 | 0 | 0 | 0 |
| speak | 0 | 0 | 1 | 0 | 0 | 0 |
| choose | 0 | 0 | 0 | 0 | 1 | 0 |
| taking fieldnotes & interviewing | 0 | 0 | 1 | 0 | 0 | 0 |
| reduce (stereotypes) | 0 | 0 | 0 | 0 | 1 | 0 |
| write/plead | 0 | 0 | 0 | 0 | 0 | 1 |
| define/analyse/answer | 0 | 0 | 0 | 1 | 0 | 0 |
| criticize | 0 | 0 | 0 | 0 | 1 | 0 |
| further (research, analyical and writing skills) | 0 | 0 | 0 | 0 | 0 | 1 |
| reinforce (opinion) | 0 | 1 | 0 | 0 | 0 | 0 |
| use knowledge | 0 | 0 | 1 | 0 | 0 | 0 |
Graphically
Across all courses
Breakdown by Level
Breakdown by Concentration
Quality of ILOs
According to the UM framework of Constructive Alignment at Course Level, “an ILO is well-formulated when a student who reads the ILO knows what to do and how well to do it in order to achieve the ILO.” (https://constructivealignment.maastrichtuniversity.nl/ilos-course-level/) For which an ILO must be: 1) Fromulated from the student’s perspective 2) Clarify the type of knowledge 3) Clarify the topics that will be taught/learned 4) Clarify the level of understanding or performance (and how this achievement is displayed) of each chosen topic.
Taking this into account the following criteria were developed to evaluate the quality of ILOs:
- Formulated from student perspective (0,1):
- Concise (-1, 0 , 1) The ILO has a single cognitive process or a single knowledge dimension, or there is a clear strong relation between them.
- Clear Capacity (-1, 0 , 1) Clear what student needs to do, (ideally it is measurable/observable), it should be fairly easy to move from this to assessment
- Clear Content (-1, 0 , 1) Content is clear and has specific terminology that could be identified by someone in the field who does not teach the course.
- Measurable behaviour (0, 1) The cognitive process dimension is observable. [TO BE FINISHED]
- Phrasing (-1, 0 , 1) Clear concise and simple phrasing with no spelling or grammatical errors.
Note:Clear Capacity was separated from measurable behaviour to allow for verbs such as “understand” to be included in clear capacity, however it is not obvious how moves from there to assessment, which would be if observable behaviours where used.
| Course | ILO | ranking | Concise | Clear_capacity | Clear_content | Phrasing | Student_perspective |
|---|---|---|---|---|---|---|---|
| HUM1012 | To test this theory on songs as well as poems | 5 | 1 | 1 | 1 | 1 | 1 |
| HUM1012 | To integrate gender and other axes of difference into the study of the lyric | 5 | 1 | 1 | 1 | 1 | 1 |
| HUM1012 | To express your ideas about poems and songs | 5 | 1 | 1 | 1 | 1 | 1 |
| HUM1012 | To write a poem or song, thereby experiencing the creative process that is involved first-hand | 5 | 1 | 1 | 1 | 1 | 1 |
| HUM1014 | pinpoint key characteristics of movements and developments in western literature since the 1850s, such as realism, modernism, and post-modernism. | 5 | 1 | 1 | 1 | 1 | 1 |
| HUM1014 | explain/understand several key principles and concepts of literary studies, including the concepts narrative, author, character, reader, plot, exposition, causal events, climax, denouement, narration, focalization. | 5 | 1 | 1 | 1 | 1 | 1 |
| HUM1014 | Set up a small research project within the field of literary studies under supervision of the tutor | 5 | 1 | 1 | 1 | 1 | 1 |
| HUM1014 | write a (comparative) research paper about literature | 5 | 1 | 1 | 1 | 1 | 1 |
| HUM2013 | 3. To learn how to write an art review. | 5 | 1 | 1 | 1 | 1 | 1 |
| HUM2014 | 2. To know and understand what is meant by modern and postmodern, and be able discuss this in relation to various philosophical positions dealt with in the course | 5 | 1 | 1 | 1 | 1 | 1 |
| HUM2014 | 3. To be able to trace and explain similarities and crucial differences between the various philosophical positions discussed | 5 | 1 | 1 | 1 | 1 | 1 |
| HUM2014 | 5. Be able to evaluate the theories and ideas in critical arguments. | 5 | 1 | 1 | 1 | 1 | 1 |
| HUM2031 | 3. To select the appropriate theories and methodological tools for analysis that best suits the material and argument. | 5 | 1 | 1 | 1 | 1 | 1 |
| HUM2031 | 6. To recognize the interdisciplinary nature of visual culture in its historical and contemporary overlap with scientific, artistic, and economic imaginaries. | 5 | 1 | 1 | 1 | 1 | 1 |
| HUM2043 | To learn to write a critical and informed film analysis essay. | 5 | 1 | 1 | 1 | 1 | 1 |
| HUM2046 | To explain the differences between technological determinism and the Science and Technology Studies STS approach | 5 | 1 | 1 | 1 | 1 | 1 |
| HUM2046 | To explain the methodological approaches in STS such as SCOT and Actor-Network Theory | 5 | 1 | 1 | 1 | 1 | 1 |
| HUM2056 | To analyze debates connected to contested memorial monuments, competing victim memories and “multidirectional memory” (Rothberg) | 5 | 1 | 1 | 1 | 1 | 1 |
| HUM3040 | 3. To explain how multiple identities and experiences of difference and inequality interact by adopting intersectionality as a critical theory and method. | 5 | 1 | 1 | 1 | 1 | 1 |
| HUM3040 | 4. To apply the analytical and critical skills needed to examine the dynamics through which identity and difference, inclusion and exclusion, equality and inequality are continuously produced and reproduced. | 5 | 1 | 1 | 1 | 1 | 1 |
Examples of Rephrasing ILOs
Bellow some examples of how ILOs could be rephrased:
| Course | ILO | ILO_new | Changed |
|---|---|---|---|
| COR1002 | To know the major approaches in the philosophy of science, such as the traditional or received view, Karl Poppers critical rationalism and its variants, Kuhns theory of scientific revolutions | explain and compare the major approaches in the philosophy of science, such as the traditional or received view, Karl Poppers critical rationalism and its variants, Kuhns theroy of scientific revolutions. | 1 |
| COR1002 | To have knowledge of the major problems or topics in the philosophy of science, such as the demarcation between science and non-science, the role of observation in science, the nature of scientific method notably induction and falsification, the issues of realism and instrumentalism, the problem of progress in knowledgethe difficulties regarding causality in history | explain and discuss the major problems or topics in the philosophy of science, such as the demarcation between science and non-science, the role of observation in science, the nature of scientific method notably induction and falsification, the issues of realism and instrumentalism, the problem of progress in knowledgethe difficulties regarding causality in history | 1 |
| COR1002 | To have knowledge of a number of specific problems in the foundations of the social sciences, such as explanation vs understanding, structuralist vs individualist views on explaining human behavior, the limits of prediction in the social sciences, the role of social science in society | explain and discuss a number of specific problems in the foundations of the social sciences, such as explanation vs understanding, structuralist vs individualist views on explaining human behavior, the limits of prediction in the social sciences, the role of social science in society | 1 |
| COR1002 | NA | take a position on these issues and perspectives and argue for its adoption | 1 |
| COR1003 | To understand the main trends in politics, demography, society and culture since 1945, and will be able to see and put these trends in a global context. | understand the main trends in politics, demography, society and culture since 1945, and will be able to see and put these trends in a global context. | 0 |
| COR1003 | To develop a critical attitude towards the interpretation of historical data and processes | question and critique the interpretion of historical data and processes | 1 |
| COR1003 | To Develop a critical understanding concerning the relation between perspective bias, facts, and context, as well as the difficulties regarding causality in history. | discuss the relation between perspective bias, facts, and context, as well as the difficulties regarding causality in history. | 1 |
| COR1004 | To provide the students with a basic understanding of what political philosophy is about and why it is important and useful in understanding contemporary democratic societies. | understand what political philosophy is about and use it to examine political debates in contemporary democratic societies. | 1 |
| COR1004 | To understand the central concepts like justice and equality in theory, and in application. | understand the central concepts like justice and equality in theory, and in application. | 0 |
| COR1005 | To offer a broad overview of scientific models and modelling techniques in different disciplines | recall a variety of basic scientific models and basic modelling techniques, and identify the use and purpose of models and modelling techniques in different disciplines. | 1 |
| COR1005 | To teach students how to work with models in different academic fields | use models to understand phenomena in different academic fields and their every-day-life. | 1 |
| COR1005 | To teach students how to model a specific phenomenon by using general models and modelling techniques | model a specific phenomenon using general models and modelling techniques | 0 |
| COR1005 | NA | identify the assumptions of basic models and adapt them to a different context or discipline of interest | 1 |
| HUM1003 | To acquaint students with some of the foundational texts and formative debates that have shaped Cultural Studies as an academic field of inquiry. | discuss foundational texts andformative debates that have shaped Cultural Studies as an academic field of inquiry. | 1 |
| HUM1003 | To familiarize students with key concepts, themes, and topical debates within contemporary Cultural Studies. | explain key concepts, themes and topical debates within contemporary Cultural Studies | 1 |
| HUM1003 | To introduce students to some of the central theoretical approaches within Cultural Studies, including critical theory, semiotics, material culture studies, gender theory, and critical posthumanism. | explain and compare central theoretical approaches within Cultural Studies, including critical theory, semiotics, material culture studies, gender theory, and critical posthumanism in the analysis of culture. | 1 |
| HUM1003 | To provide students with a variety of interdisciplinary critical perspectives to examine the diverse and often contested meanings of cultural objects and processes. | examine diverse and often contested meanings of cultural objects and processes from a cariety of interdisciplinary critical perspectives. | 1 |
| HUM1003 | To teach students to develop students own analyses of culture through oral and written modes of communication. | develop their own analyses of culture through oral and written modes of communication | 0 |
| HUM1007 | 1. To familiarize students with the basic notions required for any serious philosophic discussion | follow serious philosophical discussions. | 1 |
| HUM1007 | 2. To familiarize students with the basics of valid and invalid argumentation | recognise basic valid and invalid forms of argumentation, construct valid arguments and refute invalid ones. | 1 |
| HUM1007 | 3. To introduce students to some of the decisive developmental moments in the history of philosophy | recall some of the decisive developmental moments in the history of philosophy (e.g. Descartes, Hume, Kant), and explain why they were decisive | 1 |
| HUM1007 | NA | identify and explain basic problems from epistemology, logic, axiology, and ethics, recall different solutions, and evaluate said solutions. | 1 |
| HUM1010 | 1. To provide students with a better notion of law as a harmonising phenomenon in European culture | identify and explain the harmonising effect of law in European culture | 1 |
| HUM1010 | 2. To provide students with a basic notion of similarities and differences in the approach to law in the various member states of the European Union and the USA | compare different approaches to law from the various member states of the European Union and the USA | 1 |
| HUM1010 | 3. To give students a better understanding of basic legal notions such as property, contract and delict | explain basic legal notions such as property, contract and delict. | 1 |
| HUM1010 | 4. To provide students with a greater ability to evaluate the significance of the transfer of law making powers from the national to the European institutions | evaluate the significance of the transfer of law making powers from the national to the European institutions | 0 |
| HUM1011 | To provide students with an advanced introduction to the diverse domain of the arts such as painting, literature, music and performance art. | NA | 0 |
| HUM1011 | To broaden the students theoretical understanding of art. | NA | 0 |
| HUM1012 | To explore the theory of the lyric | To explore the theory of the lyric | 0 |
| HUM1012 | To test this theory on songs as well as poems | To test this theory on songs as well as poems | 0 |
| HUM1012 | To integrate gender and other axes of difference into the study of the lyric | To integrate gender and other axes of difference into the study of the lyric | 0 |
| HUM1012 | To be able to analyze songs and poems in a sophisticated way, to discuss lyrical texts and songs systematically | analyze songs and poems in a sophisticated way, to discuss lyrical texts and songs systematically | 0 |
| HUM1012 | To express your ideas about poems and songs | To express your ideas about poems and songs | 0 |
| HUM1012 | To become familiar with a number of classic Anglo-American poems | To become familiar with a number of classic Anglo-American poems | 0 |
| HUM1012 | To know at least one poem or song by heart, and to know how to recite/sing it | memorize at least one poem or song by heart, and to know how to recite/sing it | 1 |
| HUM1012 | To write a poem or song, thereby experiencing the creative process that is involved first-hand | To write a poem or song, thereby experiencing the creative process that is involved first-hand | 0 |
| HUM1013 | 1. To provide a basic overview of the development of the idea and identity of Europe. | trace the development of the idea and identity of Europe | 1 |
| HUM1013 | 2. Highlighting the specific characteristics of Europe s history, notably in comparison with that of other non-European societies | Identify and compare specific characteristics of Europe’s history with those of non-European societies that led to the “Europen miracle” and Europes separate identity. | 1 |
| HUM1013 | 3. Confronting students with some of the basic theoretical and methodological problems in dealing with this subject matter. | understand the basic theoretical and methodological problems in dealing with this subject matter | 1 |
| HUM1013 | NA | question and evaluate discourse on Europe, European identity, and identity in general | 1 |
| HUM1013 | NA | judge the advantages and limitations in the study of identity ( including in the social construction of identity, the symbolic construction of community, and the notions of representation, invention of tradition, and “lieux de mémoire”). | 1 |
| HUM1014 | pinpoint key characteristics of movements and developments in western literature since the 1850s, such as realism, modernism, and post-modernism. | pinpoint key characteristics of movements and developments in western literature since the 1850s, such as realism, modernism, and post-modernism. | 0 |
| HUM1014 | explain/understand several key principles and concepts of literary studies, including the concepts narrative, author, character, reader, plot, exposition, causal events, climax, denouement, narration, focalization. | explain/understand several key principles and concepts of literary studies, including the concepts narrative, author, character, reader, plot, exposition, causal events, climax, denouement, narration, focalization. | 0 |
| HUM1014 | apply several of these literary concepts in the analysis of novels and thus to be able to analyze novels in a sophisticated way and to discuss literature systematically. | apply several of these literary concepts in the analysis of novels and thus to be able to analyze novels in a sophisticated way and to discuss literature systematically. | 0 |
| HUM1014 | Set up a small research project within the field of literary studies under supervision of the tutor | Set up a small research project within the field of literary studies under supervision of the tutor | 0 |
| HUM1014 | write a (comparative) research paper about literature | write a (comparative) research paper about literature | 0 |
| HUM2003 | To acquaint students with historical configurations and intersections of race, class, gender and sexuality, and the way in which they were conceptualised and sometimes newly invented in science, philosophy and social theory. | discuss historical configurations and itersections of race, class, gender and sexuality, and recognise the way in which they were conceptualised and sometimes newly invented in science, philosophy and social theory. | 1 |
| HUM2003 | To acquaint students with the way in which these configurations of difference have structured cultural texts and images, individual identities and organisations. | examine the way in which these configurations of difference structure cultural texts and images, individual identities and organisations. | 1 |
| HUM2003 | To acquaint students with the way in which such intersecting categories of difference have constituted and still constitute inequalities and differences of power, resulting in invisibility, restricted access to sources etc. | examine the way in which such intersecting categories of difference have constituted and still constitute inequalities and differences of power, resulting in invisibility, restricted access to sources etc. | 1 |
| HUM2003 | To introduce “gender” as a useful category in historical analysis | use “gender” as a category in historical analysis. | 1 |
| HUM2003 | To introduce students to Michel Foucault’s “History of Sexuality” and “discourse analysis” and to introduce students to the “cultural work” of literature within the discourse of colonialism (Joseph Conrad) and to train students to connect theories of gender and diversity to historical case studies | connect theories of gender and diversity to historical case studies | 1 |
| HUM2003 | “” | explain the essence of Michel Foucault’s “History of Sexuality,”discourse analysis“, and the”cultural work" of literature within the discourse of colonialism (Joseph Conrad) | 1 |
| HUM2003 | To get aquainted with the theory of intersectionality | use the theory of intersectionality in their analyses of social phenomena | 1 |
| HUM2003 | NA | reflect on how categories of difference play a role in their everyday life | 1 |
| HUM2005 | 1. To provide students with a historical and philosophical overview of these two cultural periods (and worldviews), using an interdisciplinary and topic-oriented approach, based on an opposition of the two above-mentioned periods and views | compare the worldviews of enlightment and romanticism, as well as their historical contexts. | 1 |
| HUM2005 | 2. Students should also gain insight in the relationship between social, literary, artistic and philosophical developments in eighteenth and nineteenth century Europe, and get an idea of the far-reaching influence that Romanticism and the Enlightenment, as well as the romantic criticism of the Enlightenment, have had on art, science, and political discussions up to today. | explain the relationship between social, literary, artistic and philosophical developments in eighteenth and nineteenth century Europe | 1 |
| HUM2005 | “” | identify the influence that Romanticism and the Enlightment, as well as the romantic criticism of the Enlightment, have had on art, science, and political discussions up to today | 1 |
Methods in ILO’s
A separate question to consider is: how many of our ILOs explicitly refer to methodology? During the verb extraction, a comment was also made refering to whether or not the ILO explicitly refered to methodology. Out of the 625 ILOs existing in the curricula, there were 15 about methodoloy (this is 2.4%).
Here are the ILO’s which I marked as refering to methodology, the verbs used have been added for completeness:
| Course | ILO | Verbs_used |
|---|---|---|
| HUM1013 | 3. Confronting students with some of the basic theoretical and methodological problems in dealing with this subject matter. | [get] overview |
| HUM1014 | write a (comparative) research paper about literature | write, compare |
| HUM2031 | 3. To select the appropriate theories and methodological tools for analysis that best suits the material and argument. | select, analyse |
| HUM2046 | To explain the methodological approaches in STS such as SCOT and Actor-Network Theory | explain |
| HUM3034 | To familiarize students with theory, historiography and methodology of historical sociology, combining a macro historical and a macro sociological approach; | [gain] familiarity |
| HUM3050 | To understand and be capable of recognizing different methods that are implemented in aging research, such as visual methods, narrative and biographical methods, and ethnographic approaches. | understand, recognize |
| SCI3046 | 2. To learn which methods a brain researcher can use to investigate the neuronal bases for different mental processes. | learn |
| SSC1005 | To gain a basic understanding of the methods of psychological research. | understand |
| SSC2008 | 1. Knowledge of general developmental issues with respect to theories and experimental design. | know |
| SSC2050 | 2. To be able to describe methods and tools typically used in this field and experiments have have been conducted. | describe |
| SSC2059 | 4. To gain familiarity with methodological tools utilized in the analysis of social movements | [gain] familiarity |
| SSC3002 | 2. To provide students with a basic understanding of the key concepts, approaches, models and methods of public policy analysis & evaluation. | understand |
| SSC3017 | 4. Conduct primary research and analyse primary and secondary data in the field of social and environmental entrepreneurship | (conduct) research, analyse |
| UGR2001 | 4. To select and applying relevant research methodologies accordingly | select, apply |
| UGR3001 | 4. To select and applying relevant research methodologies accordingly | select, apply |
Test
## # A tibble: 627 x 21
## Course ILO `Course_Intenti~ `Active/Impicit` `Methodology LO`
## <chr> <chr> <chr> <lgl> <chr>
## 1 COR10~ To k~ S NA <NA>
## 2 COR10~ To h~ S NA <NA>
## 3 COR10~ To h~ S NA <NA>
## 4 COR10~ To u~ S NA <NA>
## 5 COR10~ To d~ S NA <NA>
## 6 COR10~ To D~ S NA <NA>
## 7 COR10~ To p~ C NA <NA>
## 8 COR10~ To u~ S NA <NA>
## 9 COR10~ To o~ C NA <NA>
## 10 COR10~ To t~ C NA <NA>
## # ... with 617 more rows, and 16 more variables: `Methodology
## # mentioned` <chr>, Verb_used_1 <chr>, Verb_used_2 <chr>,
## # Verd_used_3 <chr>, `Alternative capacity` <chr>, Concise <dbl>,
## # Clear_capacity <dbl>, Clear_content <dbl>, Phrasing <dbl>,
## # Measurable_behaviour <dbl>, Total_Rank <dbl>, `Sofia is not
## # sure` <dbl>, Reason <chr>, Grammas_spelling <dbl>, Level <chr>,
## # Concentration <chr>Bibliography
Anderson, L., Krathwohl, D., & Bloom, B. (2001). A taxonomy for learning, teaching, and assessing: a revision of Bloom’s taxonomy of educational objectives (Complete ed.). New York: Longman.